Skip to content

HADOOP-17483. Magic committer is enabled by default.#2656

Merged
steveloughran merged 1 commit intoapache:trunkfrom
steveloughran:s3/HADOOP-17483-magic-committer-on-lean
Jan 27, 2021
Merged

HADOOP-17483. Magic committer is enabled by default.#2656
steveloughran merged 1 commit intoapache:trunkfrom
steveloughran:s3/HADOOP-17483-magic-committer-on-lean

Conversation

@steveloughran
Copy link
Contributor

@steveloughran steveloughran commented Jan 27, 2021

  • core-default.xml updated
  • CommitConstants updated
  • All tests which previously enabled the magic commiter now rely on
    default settings. This helps make sure it is enabled.
  • Docs cover the switch, mention its enabled and explain why you may
    want to disable it.

Change-Id: I40a24a34d519f412d5669ec7ca1de813ad071625

@steveloughran
HADOOP-17483. Magic committer is enabled by default.
8596209 
* core-default.xml updated
* CommitConstants updated
* All tests which previously enabled the magic commiter now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.

Change-Id: I40a24a34d519f412d5669ec7ca1de813ad071625
@steveloughran steveloughran added the fs/s3 changes related to hadoop-aws; submitter must declare test endpoint label Jan 27, 2021
@steveloughran
Copy link
Contributor Author

steveloughran commented Jan 27, 2021

Tested: S3 london -Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep

Unbuffer test failure; also triggered a related failure in ITestS3AContractStreamIOStatistics

Filed: https://issues.apache.org/jira/browse/HADOOP-17499

These are network buffer related; read() calls returning less than the full buffer.

@steveloughran steveloughran mentioned this pull request Jan 27, 2021
Closed
liuml07
liuml07 approved these changes Jan 27, 2021
View reviewed changes
Copy link
Member

@liuml07 liuml07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

the change is indeed simpler than #2637

@steveloughran steveloughran merged commit 0bb52a4 into apache:trunk Jan 27, 2021
asfgit pushed a commit that referenced this pull request Jan 27, 2021
@steveloughran
HADOOP-17483. Magic committer is enabled by default. (#2656)
2d124f2 
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.
@steveloughran
Copy link
Contributor Author

steveloughran commented Jan 27, 2021

thanks. merged to 3.3+

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 27, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 34m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 8 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 8s Maven dependency ordering for branch
+1 💚 mvninstall 20m 31s trunk passed
+1 💚 compile 20m 31s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 compile 17m 44s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 checkstyle 3m 49s trunk passed
+1 💚 mvnsite 2m 26s trunk passed
+1 💚 shadedclient 20m 11s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 42s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 2m 21s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+0 🆗 spotbugs 1m 17s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 34s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 1m 26s the patch passed
+1 💚 compile 19m 47s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javac 19m 47s the patch passed
+1 💚 compile 17m 54s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 javac 17m 54s the patch passed
-0 ⚠️ checkstyle 3m 48s /diff-checkstyle-root.txt root: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28)
+1 💚 mvnsite 2m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 shadedclient 13m 33s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 40s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 2m 17s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 findbugs 3m 48s the patch passed
_ Other Tests _
+1 💚 unit 17m 48s hadoop-common in the patch passed.
+1 💚 unit 2m 3s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 58s The patch does not generate ASF License warnings.
228m 42s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2656/1/artifact/out/Dockerfile
GITHUB PR #2656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint
uname Linux c4bfbc05866f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 7c4ef42
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2656/1/testReport/
Max. process+thread count 2891 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2656/1/console
versions git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran deleted the s3/HADOOP-17483-magic-committer-on-lean branch February 11, 2021 15:14
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
@steveloughran
CDPD-22826. HADOOP-17483. Magic committer is enabled by default. (apa…
9c538bd 
…che#2656)

* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.

this patch updates core-default.xml; the 7.1-maint patch did not as
the magic committer was already enabled there.

Change-Id: I409a6c444fae843046a700a5f9e645033bdb411a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liuml07 liuml07 liuml07 approved these changes

Assignees

No one assigned

Labels

fs/s3 changes related to hadoop-aws; submitter must declare test endpoint

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@steveloughran @hadoop-yetus @liuml07